TPBF: Two-Phase Bloom-Filter-Based End-to-End Data Integrity Verification Framework for Object-Based Big Data Transfer Systems

Computational science simulations produce huge volumes of data for scientific research organizations. Often, this is shared by centers distributed geographically storage and analysis. Data corruption in the end-to-end route transmission one major challenges distributing geographically. End-to-end integrity verification therefore critical transmitting such across effectively. Although several techniques currently exist, most have a significant negative influence on rate as well overhead. Therefore, existing are not viable solutions high performance computing environments where it very common to transfer centers. In study, we propose two-phase Bloom-filter-based framework object-based big systems. The proposed solution effectively handles errors reducing memory overhead minimizing impact overall rate. We investigated memory, storage, overheads performance. experimental findings showed that suggested had 5% 10% total usage, respectively. However, observed savings terms requirements, when compared with state-of-the-art solutions.